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FOREWORD 

Joint degree programmes have been on top of the ministerial agenda from the beginning 
of the Bologna process. Although several projects at the European level have been 
carried out to analyse the specific challenges related to the quality assurance of joint 
programmes and to develop methodologies for their evaluation, no specific European 
quality assurance approach has been formally implemented yet. 

In view of the Bologna ministerial conference to be held in April 2012 in Bucharest, 
ENQA organised a seminar in September 2011 on Quality Assurance of Joint 
Programmes in collaboration with the Austrian Accreditation Council (OAR). The 
purpose of this seminar was twofold: first, to analyse the experience already gained in 
quality assurance of joint programmes as regards specific issues and methodological 
approaches, and second, to make recommendations to the ministers responsible for 
higher education in the European Higher Education Area. 

This report highlights the results of three projects, undertaken since 2004 and 
presented at the ENQA seminar in Vienna, to illustrate the width of experience that 
quality assurance agencies have in the quality assurance of joint programmes. The 
articles clearly show that what remains the major impediment for the agencies is not 
the conduct of QA of joint programmes in itself, but rather the national regulations for 
approval of the different joint programmes. 

To overcome the difficulties created by the different national regulations for approval 
of joint programmes, ENQA urges the ministers to authorise a specific European 
accreditation approach for joint programmes to be applied to all those joint programmes 
that are subject to compulsory programme accreditation at the national level. Such an 
approach would undeniably constitute a major step forward in the evaluation of joint 
programmes and bolster the efforts made so far in this particular domain of quality 
assurance in higher education. ENQA would be honoured to be invited to play an 
active role in the endeavour of developing a European approach to joint programme 
accreditation. 


Achim Hopbach 

President 

European Association for Quality Assurance in Higher Education (ENQA) 
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INTRODUCTION 

Joint degree programmes, as a means to facilitate mobility of students, have been a 
priority in the Bologna process agenda and were given increasing attention in the past 
10 years. Their implementation was raised at most of the ministerial conferences since 
the Bologna Declaration and followed up in the stocktaking exercises of 2007 and 
2009. According to the ministers responsible for Higher Education in the EHEA, such 
programmes should become more common practice. 

Despite the great challenge to evaluate joint programmes due to their intrinsic nature, 
the European quality assurance agencies strove in the last years for analysing the issues 
of quality assurance in joint degree programmes and developing specific methods for the 
external QA of such programmes. 

The most important projects in this field, most of them presented in this publication, 
are the Transnational European Evaluation Projects I and II (TEEP I and II) conducted 
by ENQA, Joint Master's Programmes - Joint Evaluations: A Nordic Challenge by the 
Nordic Quality Assurance Network, and the work done by the European Consortium 
for Accreditation, not the least in the field of mutual recognition. In addition, it is worth 
mentioning the European Master's New Evaluation Methodology (EMNEM) developed by 
EUA. 

These initiatives demonstrate that quality assurance agencies can rely on ample 
experience. However, much still remains to be done in the field of QA of joint 
programmes to overcome the numerous problems related to accreditation and 
recognition hampering the further development of joint programmes. The main challenge 
lies in the differences between national systems and legal frameworks regarding the 
approval of programmes. This does not count for national specifications in the field of 
quality assurance processes which are generally in line with the Standards and Guidelines 
for Quality Assurance in the European Higher Education Area and, thus, do not cause 
difficulties. 

In facing up to this challenge, it is important to agree upon the essential elements 
needed for the approval and mutual recognition of joint programmes and avoid 
unnecessary duplication regarding the quality assurance procedures. 

This report gives advice concerning the methodology to adopt for QA of joint 
programmes and the aspects that need to be taken into account when planning joint 
evaluations. 

The projects presented in this report show that a common European accreditation 
mechanism for joint programmes is necessary and possible (in those countries were 
compulsory accreditation schemes are in place), and that the design of transnational 
quality assurance methodologies is perfectly viable. Building a common approach would 
certainly be highly beneficial for all stakeholders involved. Indeed, the joint programme 
sector offers a unique platform to further boost the development of the EHEA. This 
objective can be achieved only if areas of common legislation at the European level are 
identified in order to facilitate the evaluation processes. 
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CHAPTER 1: 

A RETROSPECTIVE VIEW OE THE 
TRANSNATIONAL EUROPEAN 
EVALUATION PROJECT (TEEP II) 
AND IDENTIEYING OPPORTUNITIES 
FORTHEEUTURE 

Josep Grifoll, Catalan University Quality Assurance Agency (AQU Catalunya) 

This article offers a retrospective focus on the TEEP II project^ on quality assurance of 
Erasmus Mundus joint programmes launched by ENQA in 2004. The TEEP II project 
was coordinated by the Swedish National Agency for Higher Education (Hogskoleverket) 
and involved five other QA agencies, NVAO, QAA, HAC, CNE and AQU Catalunya, 
plus one representative from the EUA and the ENIC-NARIC network. The methodology 
followed the framework set out by the Bologna process which had been developed 
up until that time, together with the experience of external quality assurance in each 
of the six countries. At that time, very little experience of external quality assurance 
had been shared at the international level. What was the added value of the project in 
terms of methodology? This was clearly the tools that were incorporated to evaluate 
the effectiveness of the culture of jointness in the higher education sector. Part of the 
experience gained from this project would also presumably have served in the process of 
drawing up the ESG. 

FRAMEWORK AND METHODOLOGY DEFINITION 

It is important for the reader to question what a "joint programme" exactly means. 
Answering this is crucial, especially to understand how a quality assurance process for 
such programmes is structured and organised. 

In the development of the TEEP II project, the definition of joint programme given in 
the European Union's Erasmus Mundus programme was used. These are high quality 
integrated master's programmes offered by international consortia of universities with 
cross-border student mobility^. 

The benefit of the TEEP II project is that, through the framework of Erasmus Mundus, 
one QA methodology can be applied to three very different master's programmes. 
Although they all share the idea of interdisciplinarity as an intrinsic value of quality, the 
three programmes have different subject areas: CQMUNDUS^ (Media, Communication 
and Social Studies), EMLE"' (Law and Economics), and EURQAQUAE^ (Hydro-informatics 
and water management). 

The three programmes also had a different ECTS credit volume: 90 ECTS for 
CQMUNDUS, 60 ECTS for EMLE and 120 ECTS for EURQAQUAE. In other words, TEEP 
II is capable of formulating one methodological proposal that is applicable to several 
configurations. This is highly interesting as one of the barriers to the development of full 

1 http;//www.enqa.eu/projectitem.lasso?id=108&cont=pastproj Detail 

2 http;//eacea. ec.europa.eu/erasmus_mund us/program me/actionl_en.php 

3 http;//www.com undus.net 

4 http://www.emle.org 

5 http://www.euroaquae.eu 
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European integration in the quality assurance and public recognition of higher education 
lies in the differences in national regulations and requirements concerning structural 
aspects of degree programmes, for example the number of ECTS. 

It should be emphasised that the TEEP II project was based on the assumption that 
the Erasmus Mundus master's programmes were selected according to a process in 
which the keynote was the high quality of the proposals. On the basis of this premise, 
the objective of the methodological development of the TEEP II project was to promote 
institutional debate on the concept of "jointness" and interaction among reviewers in 
order to identify areas for quality enhancement. This orientation was in fact necessary 
given that, at the time when TEEP II was being implemented, the masters' proposals were 
undergoing an international review for the very first time. 

The methodology designed in TEEP II is compact and consistent; it provides relevant 
quantitative and identification information on the programme and then looks into how 
the mechanisms for cooperation, the level of programme (descriptors) and its quality 
assurance system are developed. 

METHODOLOGY AND POSSIBLE TOOLS FOR THE FUTURE 

The analysis of this methodology as applied to TEEP II should enable us to verify its 
viability as a starting point for a comprehensive proposal at European level. 

Firstly, in relation to public information on the programme, all Internet-related tools 
need to be upgraded, in particular the websites of the teaching consortia, which need 
to set out quality information on the programme. One can only speculate on the need 
for these websites, aside from providing information on the academic orientation of the 
programme, to provide up-to-date quantitative figures on the profile and activities of 
the students and teachers. Could this be a platform for testing a system of European 
indicators? 

Cooperation mechanisms make up the second part of the TEEP II evaluation 
methodology. Without going into details about the cooperation mechanisms that are 
fundamental in the management of joint programmes, such as the arrangements for 
student mobility within the programme or the language policy, TEEP II deals with aspects 
linked to resource management and funding that may well call for good practices 
on incentives. The existence of high quality joint European programmes, as set out 
in the objectives of Erasmus Mundus, requires adequate public funding that is often 
considerable. The sustainability of these programmes lies in the additional provision of 
resources. In other words, should joint European programmes that are not of high quality 
be funded? Can we transform this question into a pilot test evaluation aimed at making 
this high quality more transparent? 

The academic level and content of programmes form the third part of the TEEP II 
methodology. This includes, in close connection with the cooperation mechanisms, 
the evaluation of what is a programme's added value, a point which requires ex-ante 
evaluation, plus evaluation whilst the programme is running. The concept of added value 
is a tricky one; one can of course refer to the availability of a new product or service that 
complements the existing provision, but a programme should clearly be evaluated on 
the basis of its impact. This brings us back to the question raised above of the excellence 
of this type of programme. I am not talking about excellence based on compliance with 
standards - ideally, these joint European programmes should anticipate the development 
of new standards in higher education in the coming years, especially with regard to the 
defining and achievement of learning outcomes. 
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Lastly, the TEEP II methodology incorporates a section on quality assurance where 
the culture of jointness is discussed. University consortia that manage joint programmes 
need to integrate quality assurance practices so that only one system is established. The 
approach that is introduced in TEEP II of encouraging common structures for quality 
assurance opens up an entirely new dimension if it is developed appropriately. In fact, it 
paves the way to simplification of the external review and the possibility of international 
review. 

These are interesting lessons learned that can enable progress to be made in the 
framing of European policy for the quality assurance of joint programmes. 

The quality assurance methodology needs a particular approach to deal with the 
constituent processes of joint programmes. The ex-ante evaluation process will obviously 
need to be rigorous, although one should bear in mind that planning is one thing, but 
the programme being actually up and running is another: the time frame between the 
two is long. Right at the very beginning, when the programme starts to run, it is highly 
desirable to have mechanisms in place that help to identify enhancement opportunities 
and to implement corresponding measures. 

This lesson learned should enable a reassessment to be made of the processes of 
accreditation and the recognition of excellence during the initial stage of the programme. 

When joint programmes were evaluated in the TEEP II project, the use of what are 
today known as Internet-based social networks was at a very early stage. Some of the 
best known today did not even exist. It is therefore essential that the definition of quality 
for this type of transnational programme takes into account the appropriate integration 
of these online-based tools and social networks with regard to teaching and learning 
strategies and quality assurance. 

The coordination mechanisms need to be defined but, more particularly, they need 
to work. Internal and external evaluation processes need to observe what the practice 
is in this field. On the other hand, the coordination mechanisms are not just tools for 
the effective management of the programme, but also opportunities for developing 
cooperation agreements associated with the joint programme itself, for example, on the 
exchange of academic staff. Setting in place strategies of this type further increases the 
added value provided by the programme. 

Lastly, it is worth highlighting how valuable the presence of international students on 
programmes of this type is when taking into account the quality of these programmes. 
While it is common to believe that the management of different administrative matters, 
such as student accommodation and residence permits, is highly important, a good 
quality assurance system should encourage student participation in aspects associated 
with the quality of teaching in the programme. It is not of particular interest that students 
on joint programmes rate the opportunity to travel and live in other cultures as the 
main added value of these transnational programmes. In the same way, international 
stakeholders should be involved as agents who help to expand the added value of joint 
programmes, particularly in the sphere of collaboration between programmes and 
professional practice. 

CONCLUSIONS 

One of the demands made by the participating institutions at the end of the review 
stage was accreditation by ENQA. At that time, the TEEP II methodology was not set up 
for this purpose, but the possibility of being able to construct a European accreditation 
mechanism for joint programmes would be highly beneficial. In order to achieve such an 
important objective, areas of common legislation at European level which facilitate the 
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evaluation and accreditation process need to be identified. Coming back to the questions 
raised earlier about the definition and justification for the existence and public funding of 
joint programmes: are external review mechanisms needed to ensure the added value of 
this type of programmes and to safeguard their subsistence? Will they be necessary for 
reinforcing their transparency or promoting excellence? 

Whatever the case, from this retrospective look at the TEEP II project, one can 
conclude that the design of transnational methodologies is perfectly viable, and that 
the joint programme sector offers an unparalleled platform to move forward in the 
development of the EHEA, and particularly for making proposals which open up new 
dimensions in quality assurance and quality enhancement in higher education. 


9 


CHAPTER 2: 

JOINT MASTER'S PROGRAMMES - 
JOINT EVALUATIONS: A NORDIC 
CHALLENGE 

Kirsi Hiltunen, Finnish Higher Education Evaluation Council (FINHEEC) 

This paper concerns lessons learned from a Nordic project "Joint Master's 
Programmes"*’, which was implemented in 2008-2009 in all Nordic countries^ under 
the auspices of the Nordic Network for Quality Assurance in Higher Education (NOQA) 
and with the participation of five Nordic QA agencies (ACE Denmark, EVA, FINHEEC, 
Hogskoleverket and NOKUT). The main purpose of the project was to develop and 
present a model or alternative models for joint evaluation of joint Nordic master's 
programmes. The project also offered an opportunity for the participating QA agencies 
to develop a profound understanding of each other's national QA criteria and methods. 
The project was financed by the Nordic Council of Ministers, and the project group 
was appointed by NQQA and involved members from all above-mentioned Nordic QA 
agencies. 

FRAMEWORK AND METHODOLOGY 

Since 2007, the Nordic Council of Ministers has had a funding instrument called "The 
Nordic Master Programme", which is considered to be the Nordic Flagship for the 
promotion of higher education in the Nordic countries, i.e. to stimulate cooperation 
between Nordic higher education institutions on the development of joint master's 
programmes of high international quality that will attract both Nordic and non-Nordic 
students. A total of 16 programmes have been selected in the three calls for applications. 
Each selected programme is allocated 1 million DKK (-135,000 €) for the development 
and implementation of the programme. Programmes have to come from a subject area 
within a Nordic area of excellence, and they have to be two-year (120 ECTS) programmes 
that result in a joint or double degree. In addition, there must be at least three Nordic 
countries involved in the consortium. It was two of these programmes that the evaluation 
project, discussed in this paper, reviewed and piloted its methodology on. The outcome 
of the project was twofold: on the one hand, a report to the Nordic Council of Ministers 
on evaluation methodology, and on the other hand, reports to the two programmes on 
the quality of their educational and QA activities. 

The project was implemented well in line with the European development. It was 
taken for granted that the general pattern of the evaluation would follow the Standards 
and Guidelines for Quality Assurance in the EHEA (ESG). Many of the points raised in 
EUA's European Master's New Evaluation Methodology and TEEP II were useful in the 
preparation of the evaluation criteria and project report. 

The project group got a mandate by the Nordic Council of Ministers to firstly 
investigate QA practices and legislation with regard to QA and joint master's 
programmes in the Nordic countries. At the beginning of the project, the project 
group also made an attempt to find out the number and types of joint Nordic master's 
programmes through a questionnaire sent to all universities in the Nordic countries, 

6 Final report (incl. a report on the pilot evaluations) is available on NOQA’s web page: www.nokut.no/noqa. 

7 Denmark, Finland, Iceland, Norway and Sweden. 
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except Iceland. The answers demonstrated that, unfortunately, there was scant 
knowledge of the situation at the institutional level. 

IMPLICATIONS OF NATIONAL OA SYSTEMS FOR JOINT PROGRAMME 
ASSESSMENT 

It was not surprising at all to find out that the general methodology applied by the QA 
agencies in the Nordic countries resemble each other. This is due partly to a tradition 
of cooperation and information-sharing among the agencies responsible for national 
QA of higher education. Naturally, the ESG have also been instrumental in this respect. 
Consequently, all five countries use similar methodology, which includes self-evaluation, 
expert panels and public reports. Reviews are in the form of programme and subject 
evaluations, thematic evaluations, quality audits, accreditation-like practices and awards 
for excellence in higher education. 

However, it was overwhelming to notice that the national QA systems seemed to be 
moving in two different directions, and in this process, very little consideration appeared 
to be taken of the development in the other Nordic countries. The different emphases 
and national requirements created problems as regards the joint programmes in the 
different countries as well as in finding common denominators for joint evaluation. In 
particular, the substantial revision of the national QA systems in Denmark and Sweden 
were complicating factors. Denmark, Sweden and to some extent also Iceland could be 
seen as "controlling" countries, whereas Finland and Norway as countries where national 
QA systems could be seen based on trust on higher education institutions and respecting 
their autonomy. 

Thus, what were the premises for joint programme assessment? Firstly, as stated 
above, there were major differences between the systems of QA in higher education 
and the systems seemed to be diverging even more, opposite of what was anticipated 
earlier e.g. by Stensaker-Dano (2006) Nordisk kvalitetssikring av hoyere utdanning. 
Arbeidsnotat 16/2006 NIFU STEP. Secondly, there was no reliable information either 
of the number or types of joint programmes on a national basis (nor at an institutional 
level) except for the Erasmus Mundus grantees. Thirdly, by 2008, there had been no 
systematic external evaluation of joint programmes in the Nordic countries. Lastly, 
legislation with regard to joint degrees varied. 

It was felt strongly that there was a great need of joint evaluation of joint Nordic 
master's programmes in order to secure the interest of students in these programmes, 
and that these evaluations should concern each programme as a unit and be carried out 
by one expert team and result in one report. It had become obvious to the project group 
that no joint evaluation and, in particular, no accreditation could take place without more 
or less far-reaching agreements between the countries and changes in legislation. But it 
was not seen as part of the project to propose such changes. 

THREE ALTERNATIVE APPROACHES 

Based on the information that was collected and on the experiences gained through the 
pilot evaluations, the project group proposed three alternative approaches for the QA of 
joint Nordic master's programmes: 

Joint evaluation by one expert team followed by joint evaluation / accreditation (if 
necessary) accepted by all national QA agencies through a process of mutual agreement 

Joint (voluntary) evaluation for the award of a Nordic quality label 

Accreditation of individual parts of the programme in each country in accordance with 
national legislation, followed by joint audit of the QA of the programme as a whole. 
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The project group tested one of the alternatives, alternative A, which is the most far- 
reaching approach and would require significant agreements and legislative amendments 
in the Nordic countries. The main advantage of this approach is that it would require only 
one process and one decision for the evaluation of one programme. It would promote 
transparency in that stakeholders, including students, institutions and future employers 
would get an overall view of the quality of the programme as a whole. It would also 
increase mutual awareness of the QA systems in the Nordic countries. But as only one of 
these approaches was tested in pilot evaluations, the project group cannot promote any 
one of them on the basis of empirical findings. 

CRITERIA FOR EVALUATION OF JOINT MASTER'S PROGRAMMES 

The project group formulated the following criteria that might be included in a Nordic 
model for evaluation of joint programmes. The criteria were also used and tested in 
the two pilot evaluations of the project. With the exception of the criteria for jointness, 
the criteria were based on those for evaluation of master's programmes in the Nordic 
countries. Criteria were divided into six thematic areas: 

1. AIMS OF THE PROGRAMME 

Aims of the programme are clearly defined in terms of learning outcomes and in line with 
the EQF. 

2. CRITERIA FORJOINTNESS 

There is a formal agreement between the participating institutions; mechanisms and 
responsibilities of cooperation spelled out and defined; support of managements; added 
value of programme; sustainable funding strategy; language policy; adequate resources 
(both infra and staff); mobility secured; students' rights secured also in other institutions; 
contacts between teachers across institutional/national boundaries; relevance and 
accessibility of information about programme to students and external stakeholders etc. 

3. LINKS BETWEEN PROGRAMME CONTENTS, TEACHING AND CURRENT 
UP-TO-DATE RESEARCH 

Programme and teaching are based on relevant, up-to-date, research-based knowledge; 
teaching and examination methods support students' attaining the goals of programme 
and contribute to students' ability, in national and international contexts, to account 
for and discuss results of research; teachers are active researchers and they possess 
pedagogical skills; there is an active and keen research environment which also provides 
PhD programmes and where students are involved etc. 

4. RELEVANCE FOR FUTURE PROFESSIONAL CAREERS 

There is labour market demand for graduates; programme makes graduates eligible 
for employment with high demands for independent work related to research and 
development or other qualified employments; regular contacts between programme and 
employers. 

5. DEPTH AND PROGRESSION 

Programme is planned and organised as a unit in which the courses contribute to the 
whole; programme relevant, up-to-date and realistic in relation to learning outcomes 
specified and length of programme; learning outcomes cover knowledge and abilities in 
the main area of study as well as generic skills; programme demonstrates progression 
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internally and in relation to relevant bachelor programmes and is based on students' 
expected prior knowledge (a first cycle degree); includes a main area of studies 
amounting to a minimum of 60 ECTS; the minimum extent of the thesis is 30 ECTS. 

6. QUALITY ASSURANCE 

Programme has a system which assures the quality of joint provision, and guarantees 
that the aims of programme are met; continuous information collection both for this 
purpose and for the further development of programme; QA practices involve students, 
staff and other stakeholders from all participating institutions; QA includes coordination 
of assessment across the whole programme; there is adequate provision for teachers' 
academic and pedagogical development; reports on results of QA activities are publicly 
available. 

EXPERIENCES ON THE PILOT EVALUATIONS 

Two pilot evaluations were implemented in 2008-2009. The programmes reviewed were 
the Nordic Master's Degree Programme in Gerontology (countries involved: Finland, 
Iceland and Sweden) and the Nordic Master's Degree Programme in Plant Pathology 
(countries involved: Denmark, Norway and Sweden). The main purposes of the pilot 
evaluations were to test and develop further the method and criteria created by the 
project group and to provide feedback to the two programmes with regard to the further 
development of the quality of their educational and QA activities. The pilot followed the 
general principles of the ESG: criteria were published and applied consistently; (subject) 
expert and student participation in the evaluation team; evaluation process included the 
use of self-evaluation, site visits and reports; it was ensured that the procedures used 
provided adequate evidence to support the findings and conclusions reached. 

In terms of testing a method and criteria for evaluation of joint programmes, the 
pilot evaluations functioned quite well. A framework had been developed although 
adjustments would have to be made regarding certain criteria. Higher education 
institutions' attitude towards the pilot evaluations were positive; pilots were seen as 
an opportunity for improvement and contributing to providing relevant feedback to the 
programmes. But even though the pilot evaluations may have been successful, several 
difficulties related to the legal situations, administrative issues and QA procedures 
remained to be solved. 

The application of the criteria to the two pilot evaluations provided a full picture of 
the programmes. The criteria proved adequate in terms of focusing on a) programme 
content, current up-to-date research and administration - results correspond with those 
from national evaluations, and b) general challenges of running joint programmes. Most 
challenges of the two programmes were related to running a joint programme. There was 
a need for: 

• clearly stated aims and learning outcomes, including a joint syllabus, language 
policy, as well as an account of the intended added value of the programme 

• a common funding strategy (resources for planning and coordinating the 
programme; resources for mobility: staff, students, management) 

• efficient study counselling and mobility plans, which take into account the needs of 
different kinds of students 

• consideration of how to handle different semester periods 

• a special attention in the social aspect and information-sharing between the 
students (e.g. a joint introductory course, a discussion area on the web) 
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• the definition of the distinct quality of graduates from joint programmes: 
accentuation of the specific skills and competences in the diploma; universities 
proposed to develop a system for keeping track of alumni in order to be able to get 
an overview of their careers and thus of the labour market available for graduates 

• a joint QA system for the programme that takes the criteria proposed by the 
project group as its point of departure, and includes e.g. shared information on 
students from application to admission, joint procedures for monitoring students' 
progression through the programme through the use of individual study plans, 
midterm evaluations, and a plan for teacher exchanges within the programmes. 

CONCLUSIONS 

To sum up, any method of QA of joint master's programmes will have to take into account 
the ESG. It is also necessary to consider the different methodologies and criteria of the 
individual countries as well as the internal QA processes of the institutions. As stated 
above, the most important differences in the Nordic countries with regard to approach 
concern the object of evaluation, namely the focus on programme (Denmark, Sweden, 
and to some extent, Iceland) and on audits of institutional QA systems (Finland and 
Norway). Qn the other hand, there are countries with a strict legislation on accreditation 
such as Denmark and, partly Sweden, and countries with the principle of full self- 
accreditation for universities such as Finland, Norway, and, partly, Sweden. Therefore, 
when planning joint evaluations, the differences in methodology (between the Nordic 
countries) call for special arrangements and negotiations at different levels - ministries, 
QA agencies and higher education institutions all have to be involved in the process. 

With certain exceptions, national authorities and even the central administration of 
higher education institutions had little awareness of the extent of international or Nordic 
collaboration in the form of joint provision of master's programmes. Thus, most of these 
programmes were not subject to external evaluation at national level. It is obvious that 
there is much to be done in the field of QA of joint programmes. 

Finally, student perspective makes it particularly important to monitor and assess the 
quality of joint programmes. Students in joint programmes face a more complex situation 
than students studying in a national context; they must adapt to different environments 
and different learning situations. 
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CHAPTER 3' 

JOINT PROGRAMMES AND 
MUTUAL RECOGNITION OF 
ACCREDITATION DECISIONS 

Mark Frederiks, Accreditation Organisation of the Netherlands and Flanders (NVAO) 

This contribution focuses on two projects regarding joint programmes carried out by the 
European Consortium for Accreditation in higher education (ECA). Within the framework 
of the TEAM 1 1 project, ECA member agencies carried out five pilot accreditation 
procedures of joint programmes during autumn 2009 and spring 2010. Based on 
these experiences ECA has proposed a way forward regarding accreditation of joint 
programmes. This proposal has materialised in the JOQAR project which has started in 
October 2010 and will run until October 2013. 

JOQAR involves the establishment of a European coordination point for external QA 
and accreditation of joint programmes. The coordination point will administer a publicly 
accessible knowledge base on joint programmes. A methodology for single accreditation 
procedures of joint programmes under guidance of the coordination point is being 
developed and this will be tested through four pilot procedures. In addition, a multilateral 
recognition agreement between agencies regarding the mutual acceptance of the results 
of accreditation of joint programmes has been developed and will be further enhanced 
and enlarged with new agencies. 

Both the TEAM II and the JOQAR projects also focus on the recognition of degrees 
awarded by joint programmes. There are many recognition problems which have been 
discussed with ENIC-NARICs and for which guidelines to encourage good practices 
will be collectively developed. However, this contribution will only focus on the QA and 
accreditation aspects of joint programmes. 

JOINT PROGRAMMES ON THE RISE 

Joint programmes have been on the European political agenda, with ever increasing 
attention, for the last decade. They are thought to stimulate main aims of the Bologna 
agreements such as mobility of staff and students, cross-border cooperation and 
promotion of European higher education. According to the Bologna Stocktaking Report 
2009® there are now about 2,500 joint programmes in Europe of which over 150 are 
Erasmus Mundus programmes supported by the European Commission. These numbers 
show that there is an increasing interest from higher education institutions to cooperate 
on offering joint programmes. 

With the increased attention on joint programmes, there is also an increased focus 
on finding purposeful methods for the external quality assurance of these programmes. 
Normally, QA and accreditation procedures are carried out by one QA agency and 
focusing on how institutions are offering programmes nationally. However, since joint 
programmes are, per definition, offered jointly by different higher education institutions it 
is necessary for agencies to make sure that the quality of the totality of the programme is 
up to standard. It is important to develop methods for evaluation or accreditation of joint 
programmes that do not involve unnecessary duplication, but rather focus on building 

8 Rauhvargers A, Deane C and Pauwels W (2009), Bologna Process: Stocktaking Report. 


15 


mutual understanding and trust. As mutual recognition is a primary aim of the European 
Consortium for Accreditation in higher education (ECA)^ it was only natural that ECA 
became active in the field of joint programmes. 

TEAM II 

ECA has, as a reaction to the increase in the number of joint programmes offered in 
Europe and as a response to the calls for more focus on external quality assurance of 
these programmes, been exploring different ways in which accreditation procedures of 
joint programmes can be carried out. 

In the period from October 2008 to September 2010, ECA partners were carrying 
out the EU funded project Transparent European Accreditation decisions & Mutual 
recognition agreements II (TEAM II). The main aims of the project were: to develop a 
European methodology for accreditation of joint programmes; to explore recognition of 
qualifications awarded by joint programmes; and, to improve and extend the website 
Qrossroads“ which contains transparent information on quality assured higher education 
programmes. 

In order to develop a European methodology for accreditation of joint programmes 
ECA member agencies coordinated five experimental single accreditation procedures 
of joint programmes during autumn 2009 and spring 2010. These would function as 
empirical material for discussions on how to carry out single accreditation procedures 
of joint programmes. After the completion of each of the procedures feedback from 
experts, coordinators of the joint programmes and representatives from the quality 
assurance agencies was gathered and analysed. A methodological report showing all the 
findings was published^E All results of the TEAM II project were presented and discussed 
at a dissemination conference in June 2010 in Graz“. 

During autumn 2009 and spring 2010, ECA members coordinated the five pilot 
accreditation procedures. In total 24 higher education institutions^^ from 12 different 
European countries and nine quality assurance agencies were involved in the procedures. 


9 http;//www. ecaconsortium.net 

10 http://www.qrossroads.eu/ 

11 Kaja Braathen , Mark Frederiks and Nick Harris (2010), How to assess and accredit joint programmes in Europe: 
Methodologies tested and proposed by the TEAM2 project. 

12 Axel Aerden, Kaja Braathen and Mark Frederiks (Eds.; 2010), Joint programmes: Too many cooks in the kitchen? The 
Challenges for Accreditation, Recognition and Transparency of Joint Programmes - A Conference publication. 

13 This only refers to the full partners of the participating joint programmes. 
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NAME OF THE 
PROGRAMME 

FULL PARTNER INSTITUTIONS* 

NEED FOR 
ACCREDITATION 

COORDINATING 
QA AGENCY 

OTHER QA 
AGENCIES 
INVOLVED 

PLACE OF SITE 
VISIT 

Research Master 
Geosciences of Basins 
and Lithosphere 
(BASINS) 

VU University Amsterdam (NL) 
Uni Bergen (NO) 
Uni Rennes 1 (FR) 

NL 

NVAO 

CTI, MAC 

Amsterdam, 

NL 

Joint European Master 
in Comparative Local 
Development (CoDe) 

University of Trento (IT) 
Corvinus Uni (HU) 
Uni Ljubljana (SI) 
Uni Regensburg (DE) 

DE,HU,SI 

MAC 

GAC, 

SOAA 

Trento, IT 

EM Master Journalism 
and Media within 
Globalisation (EMMA) 

The University of Aarhus (DK) 
School of Journalism(DK) 
Uni Amsterdam (NL) 
Uni Hamburg (DE) 
Swansea Uni (UK) 
City Uni (UK) 

DE,NL 

ZEvA 

NVAO 

Hamburg, DE 

Bachelor European 
Teacher Education 
for Primary Schools 

(ETEPS) 

University College Zealand DK) 
Stenden Uni (NL) Linneus Uni (SE) 
BuskerudUC(NO) 

NL,SE, 

DK 

NVAO 

HSV 

Leeuwarden, 

NL 

Joint European 

Deusto University (ES) 

ES, DE, 

AN EGA 

AOAS, 

Deusto, ES 

Master in International 
Humanitarian Action 

(NOHA) 

Uni Cathol.de Louvain (BE), 
Uni Bochum (DE) 
UC Dublin (IE) 
Uni Groningen (NL) 
Uppsala Uni (SE) 
LUniversite Paul Cezanne (FR) 

NL 


HSV, 

NVAO 

Louvain, BE 


Table 1: Overview of the five pilot procedures 

It was agreed that the procedures would be organised according to the following 
principles: 

• One QA or accreditation organisation would be responsible for the procedure; 

• The totality of the joint programme would be assessed in accordance with the EC A 
Principles for accreditation procedures regarding joint programmes”' 

• The accreditation frameworks of the respective accreditation organisations would 
be compared, and the differences taken into account; 

• There would be a focus on learning outcomes. 

Observers from QA and accreditation agencies of other countries involved in the 
programme could be included in the procedures. 

In practice, all the procedures followed the normal steps of an accreditation 
procedure. On a more detailed level the procedures varied according to the institutions 
and agencies involved in the procedures. 

In all the pilot procedures the scope of the self-evaluations was the totality of the joint 
programme. However, in some pilots the self-evaluations were written according to only 

14 Available for downloading on: http://www.ecaconsortium.net/main/documents/main-documents 
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one set of criteria whilst in others the criteria of all the involved agencies were compared 
and assessed. 

In most cases existing criteria functioned as the scope of assessment, while in one 
case a totally new set of criteria was developed. 

The expert panels were mostly composed jointly by the involved QA agencies, 
although it varied according to how the organisation was in practice. In all the 
assessments, a discipline-specific expert, quality-assurance and international experts, as 
well as a student were included. Additionally, an expert from the professional field was 
included in most of the procedures. 

The external assessment included in most cases only one site visit, but in one of the 
procedures there were two site visits. 

The pilots also varied when it came to the actual accreditation decisions taken. In 
most of the cases, separate (coordinated) accreditation decisions were taken based on 
the same assessment. In one case a decision taken by one agency was recognised by 
another agency, and in another case the pilot was a true pilot procedure where no formal 
decisions were taken. 

After the completion of each procedure, questionnaires were sent out to the 
coordinators of the joint programmes, the experts and the representatives from the 
quality assurance agencies. They were asked to share their experience with the various 
steps of the procedures, and to specify strengths and weaknesses. The feedback 
showed that the participants were generally content with the work of the joint expert 
panels, how the site visits were conducted and the outline of the final reports. Especially 
jointly composed expert panels were viewed as a valuable asset when assessing a joint 
programme. 

Nonetheless, the feedback also showed that there are some areas which require 
special attention when carrying out single accreditation procedures of joint programmes. 

Firstly, when more than one set of criteria was used, agreeing on the level of details 
and information required in the assessment was reported to be challenging. Variations 
in national regulations result in a situation where quality assurance agencies, to some 
extent, have to focus on different aspects of a study programme. This influences the 
interpretations of criteria and the focus of the assessment. 

Secondly, in almost all the pilots, comments were made regarding the self evaluation 
reports. In most cases these reports were too descriptive, and with an imbalance of the 
information presented about each of the participating institutions and of the various 
criteria. 

Thirdly, the composition of the expert panel can be challenging when QA agencies 
have different practice in this regard. 

Finally, there were no unified ways in which the accreditation decisions could be 
taken. The decision making is affected by national regulations and needs to be taken into 
account when planning single accreditation procedures. The importance of making sure 
that the final report fulfils the requirements of all agencies involved in the procedure was 
emphasised as this will facilitate the decision making process. 

In short, it is indeed possible to assess whether a joint programme meets the 
requirements of several quality assurance agencies through one single accreditation 
procedure. The challenge is rather to agree upon the essentials needed in such a 
procedure, and find ways within the national legal frameworks to carry out single 
accreditation procedures. As a consequence, ECA has started the JOQAR project to 
explore how these findings can be implemented into practice. 
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JOQAR 

From October 2010 to October 2013, EGA partners in cooperation with other QA 
agencies (including two in India and Colombia) and ENIC-NARICs are carrying out 
the EU funded project Joint programmes: Quality Assurance and Recognition of degrees 
awarded (JOQARy^. The overall purpose of the project is to ensure that Erasmus Mundus 
programmes, and joint programmes in general, are facilitated in two specific areas: 
accreditation and recognition. With regard to accreditation, the project seeks to promote 
single accreditation procedures through: 

• the development of a multilateral recognition agreement regarding QA and 
accreditation results, and 

• the establishment of a European coordination point for external QA and 
accreditation of joint programmes. 

Qne of the aims of EGA is to build up sufficient trust between the member agencies 
so that they can mutually recognise each other's accreditation decisions. So far only 
bilateral mutual recognition agreements between accreditation agencies have been 
signed. Since joint programmes have a wider scope and involve many different countries, 
a multilateral mutual recognition agreement specifically for joint programmes would be 
beneficial. If agencies have entered into a multilateral recognition agreement, it is a lot 
easier to facilitate that the results of a single accreditation procedure will be accepted in 
each of the countries that are part of the multilateral agreement. In December 2010 the 
first six agencies have signed such a multilateral agreement (the so called MULTRA“). 
Qbservation missions for another eight agencies are currently planned within the JQQAR 
project so that the MULTRA can be expanded with these agencies. 

Another way forward is to establish a European coordination point for external QA and 
accreditation of joint programmes. The coordination point will give coordinators of joint 
programmes the opportunity to consult a specialised information centre when they need 
to undergo external quality assurance and/or accreditation of their joint programme. 

In 2011 a survey among coordinators of Erasmus Mundus joint masters programmes 
and QA agencies showed a clear need for such a coordination point^^. A work plan to 
set up the coordination point is currently being developed. The coordination point will 
also administer a publicly accessible knowledge base which contains useful information 
for accreditation, external quality assurance and recognition of joint programmes. In 
addition, when a joint programme wants to go through a single accreditation procedure, 
the coordination point will bring agencies and the joint programme together and apply 
a methodology that enables a single procedure. This methodology is based on the 
TEAM II experiences and includes agreed common criteria and a limited set of specific 
national criteria which are needed to get the results of the single accreditation procedure 
accepted in all the countries of the joint programme. This "core+" assessment framework 
for joint programmes is in its final stage and will be subsequently tested in 2012 through 
four pilot single accreditation procedures of joint programmes. 

CONCLUSIONS 

There are still many problems hindering the further development and growth of joint 
programmes, both in the field of accreditation and recognition. Many challenges 
originate in differences between national systems and legislations. EGA as a consortium 


15 More information about the JOQAR project is available on; http://www.ecaconsortium.net/main/projects/joqar 

16 See for an explanation of the MULTRA: http://www.ecaconsortium.net/main/documents/mutual-recognition- 
agreements 

17 Axel Aerden and Kaja Braathen (2011), Feasibility study: a coordination point for joint programmes. 

The survey report can be downloaded from: http://www.ecaconsortium.net/main/documents/publications 
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of national accreditation agencies striving for mutual recognition is in an excellent 
position to help joint programmes to overcome these problems. 

The EU funded TEAM II and JOQAR projects show that a common approach is 
necessary and possible. This approach entails expanding mutual recognition agreements, 
developing a methodology for single accreditation procedures carried out by national 
agencies with the assistance of a European coordination point, and testing these in pilot 
procedures. It is supplemented by a knowledge base where institutions, agencies and 
stakeholders can find the necessary information on QA, accreditation and recognitition 
of joint programmes. 

Although practical problems facing joint programmes are manifold, the JOQAR 
project, which is now halfway, shows that pragmatic solutions are within reach. 
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RECOMMENDATIONS TO THE 
MINISTERS RESPONSIBLE FOR 
HIGHER EDUCATION IN THE EHEA 

Joint degree programmes, resulting in a truly (trans-)European degree, facilitating 
mobility and attracting students from other EHEA countries or even from outside the 
European Higher Education Area, have been on top of the agenda from the beginning of 
the Bologna process. They have already been mentioned in the Sorbonne Declaration 
(1998), and the Ministers of Higher Education involved in the Bologna process raised 
the issue at most of the ministerial conferences since then. In 2007 and 2009, the 
implementation of joint programmes was also followed up in the stocktaking exercises. 

At the Leuven/Louvain-La-Neuve conference in 2009, ministers emphasised that "joint 
degrees and programmes [...] shall become more common practice". 

Already in the early days of joint programmes, it became clear that they challenge the 
existing national quality assurance procedures: more than one provider develop and offer 
a joint programme which is studied at more than one institution in more than one country 
with different political and legal frameworks and not the least differing quality assurance 
regimes. 

The European quality assurance agencies accepted this challenge and made a great 
effort in the last years to analyse the specific issues of quality assurance in joint degree 
programmes and to develop approaches for this specific case. 

Hence, today one can rely on ample experience with quality assurance of joint 
programmes. Two main lessons learnt are as follows: 

In principle, part II of the ESG is applicable to accreditation of Joint degree 
programmes. However, national specificities in the application of the ESG, namely 
regarding the composition of expert panels, design of site visits, and the formal decision 
making constitute differences in the process design. HEI and quality assurance agencies 
developed approaches to joint accreditation procedures of the responsible agencies. 
Although joint accreditation procedures lower the burden on the HEI, they defy the 
accreditation agencies concerned because joint procedures must be designed for each 
case. An alternative would be the recognition of parts of the accreditation conducted 
by one responsible agency through another agency, which also challenge the agencies 
in terms of comparing procedural regulations and criteria. Hence, the ultimate aim is to 
reduce the accreditation of Joint degree programmes to a single procedure with effect in 
all national jurisdictions concerned. 

A commonly accepted design of external quality assurance approaches to Joint degree 
programmes is not a difficult task. Although different approaches to quality assurance 
are in place (accreditation vs. evaluation vs. audits; programme level vs. institutional 
level), agencies demonstrated in the last years that these differences can be easily 
overcome, due to the alignment of existing national procedures with the ESG. 

A more substantial hurdle for accrediting Joint degree programmes lies outside the 
remit of most of the quality assurance agencies in the EHEA, in national specifications for 
approval of degree programmes. There is clear evidence that national formal regulations 
which refer to the denomination of degrees, workload, semester periods, etc. are a much 
bigger obstacle for implementing Joint degree programmes than the accreditation or 
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external quality assurance as such. Hence, more flexibility regarding criteria for Joint 
degree programmes is paramount. 

Based on experience, ENQA recommends to ministers responsible for higher 
education in the European Higher Education Area to allow/authorise/enable a specific 
European accreditation approach for Joint programmes which should be applied to all 
those Joint programmes that are subject to compulsory programme accreditation at 
national level. ENQA would welcome the opportunity to explore the practicalities in 
developing such an approach. 

THE EUROPEAN ACCREDITATION APPROACH FOR JOINT PROGRAMMES 
SHOULD BE DESIGNED BASED ON THE FOLLOWING PRINCIPLES: 

• Accreditation of Joint programmes should be based on a commonly accepted 
definition of a Joint programme. This definition should emphasise the 'jointness' as 
specific feature of such a programme. 

• Deduced from that definition, a specific set of criteria for accrediting Joint 
programmes should be developed. This set would be based on the proper 
application of the Qualifications Framework of the EHEA, ECTS, DS and internal 
quality assurance in accordance with part I of the ESG. In addition, criteria 
regarding the specific nature of Joint programmes, namely joint responsibility, joint 
development and joint provision of the programme would be applied. 

• Additional national criteria should only be applied if they are related to the quality 
of the programme. Formal national criteria as the most important obstacle to 
implementing Joint programmes should not be applied. 

• The procedural regulations regarding the new approach should be based solely on 
part two of the ESG and guarantee especially the restriction to only one procedure 
with only one expert panel including international members and to only site visit. 

• Agencies should apply these criteria and these regulations instead of national 
formal, but not quality related, specifications (in case accreditation is mandatory) 
which means that Joint programmes that have been accredited with the European 
approach would not need to be accredited for a second time at national level. 
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ANNEX 


QUALITY ASSURANCE OF JOINT PROGRAMMES 

ENQA SEMINAR HOSTED BY OAR 

19 September 2011, Vienna, Austria 

VENUE 

Alte Kapelle 

Institut fur Ethik und Recht 
University of Vienna 
Spitalgasse 2-4, Hof 2.8 
1090 Vienna 


PROGRAMME 

9.00 Welcome and Introduction 

Achim Hopbach, ENQA President 

9.15 Quality assurance of Joint programmes 

Chair: Alexander Kohler, AQA 

Transnational European Evaluation Project II (TEEP II), 

Lessons learned 

Josep Grifoll, AQU, ENQA Board 

10.15-10.45 Coffee break 

"Joint Master's Programmes - Joint Evaluations: A Nordic Challenge 
Lessons learned 

Kirsi Hiltunen, FINHEEC 

12.00 Lunch Break 

13.30 Joint programmes and mutual recognition of accreditation 
decisions 

Mark Frederiks, ECA 

Chair: Elisabeth Fiorioli, OAR, ENQA Board 
14.30-15.00 Coffee Break 

15.00 Final Discussion: Recommendations 

16.00 Conclusions 

Elisabeth Fiorioli, OAR, ENQA Board 


THIS REPORT is based on the ENQA seminar on Quality Assurance of Joint Programmes that was held on 19 
September 2011 in Vienna, Austria. The purpose of this seminar was to analyse the experience already gained in 
quality assurance of joint programmes as regards specific issues and methodological approaches, and to make 
recommendations to the ministers responsible for higher education in the European Higher Education Area. 
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